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The Setting , . ' 

<s " A three year program to develop Alaskan Native leadership y/as funcjed 
about a year and one hfiU ago by a found^ion' grant. Its two overall and , 
complementary goals' are presently understood to be:, (i^l'meet immediate high • 
level training needs of Alaskanr Native, specifically ia-regards to the regional 
corporations as they carry out the Drovi$ions.of the J\laska Native Land Claims 
Settlement Act of 1971, and (2) .promote changes in University education which 
will allow thSse educational needs to be met long range by a more flexible, - 

irresponsive system. This project wafe originally conceived of four yeacs ago'^as 
a^ratHer simplistic plan of carrying informal ecjucation jinto outreach centers • ' 

, through pobperafive Extension.. At its time of funding it had evolved into^the 
firsi program ever jointly sponsored by the Upiversijy of Alaska and a statewide^ 

^Native organization (the Alaska Native Foundation) . • 

- * . »>'... 
. ' A Program Policy CounGil was sti^lated to consist of seven members {three 
University, three Native leadership, and one elected by th^ other six) , The 
council was to oversee and give general- directions to tjie program. The chosefi 
council members are, all in powerful positions, including the -e'xecutive vice 
president of t^he University dnd*the dirBcto^of two "University .research institutes,, 
the presidents of the Alaska Mati>>e Foundation^Cs^atewide, non-profit) and the 
/Alaska Federation of Natives (statewide,^ political), . This unique organizational 
arrangement ^between Unaver:sity and Native leadership is a prime factor in the 
evaluation 'Situation . -r' - ' 4 ^ ^ , 

*: ' . • * 

< • >. ' 

The Evaluate r * ' ^ . . ' ^ 

As an evaluation specialist at the Center' for Northern Educational -Research, 
University. of Alaska, I was contracted as a par^t-time evaluator by, and to^work 
for, th,e. policy council. Some members of tRe council felt that the process Ihe , 
program was initiating between the University ^anti Native leadership -was itself in ^ 
need of evaluation,. whereas empnasis on p^roduet could increase in later stages, . 
when training. i5rograrT^s and/or other outcomes had iD.een developed. '/ • ^ 

JflThe idea of doing a process evaluation int^^igued me. I first went back 'and 
reviewed the evaluation literature on process evaluation ix»etHbd6logy . (Proce^ ^ 
as I am ^Jiscussihg ifis concerned broadly with prograjrn, rather than its more 
restrictive use with the learninglpVocess . ) Many m^ethodologies and models 
aSdress the concept^, with slightly different names and*meanings , In*theCIPP , 
model '{S^ufflebeam, '1969) the first P is for process evaluation. The UCLA model 
(Klein, 1971)? divides this BvaluaXiOn st'age into implementation artd progress 
evaluation. Scriven (1967) talks of formative evaluation. Rippey (1973) takes a- • 



N^Ltive is legiilly defined as a person who is 5b least one-fourth Indian, Eskimo, 
or Aleut, ^ " 



slicfhtly different slant on process v/hich he calls transaction^ evaluation. The* 
I'ist goes on'but? basically all, are concerned about what is going on or happens 
• within the'imaginary black box between input and output. The basic concern is 
to collect data pn what is actually going oh, provide information for.correctiye ^ 
deciSionj-maliing acticyi during the project, and to reeord descriptive informatign 
to aid in understanding of later <)utputs. Itxan b& contrasted to summative, 
'outcome, or product evaluation. ^ . 

* . » ' * * 

Steele (1973, p. 47) in her review of evaluation approaches, cites as a 

limitation of hgr monograph: * ' . ^- " • • . ' , 

"The lack of clear procedufal directions;. In-sunimarizing ' , . ; 
't}>e approach^s>.priority has gone to the parts that outline the /. • ^ 
*what' and 'why' of 'evaluation rather thaa those' that deal with 
the 'how' of carrying out the procedure)" ' . ' ♦ 

1 believe this same shortcoming could &l'so be said' to 'characterize process ^ . 
evaluation methodology as it now exists under its varied nomenclature'. \ One can 
find a considerable amount written about what process evaluation is and why it . 
' should be done-,, but little direction as to how it should bg, done. . ^ , 

* ' ^ Prior to formal contf^icting I presented a brief paper to the Council on 
evaluation, "emphasizing the importance of studying process, describing the 
project 'anci pro^iding^data for de'cision making. 

Early Evaluation Attempts • * " . ^ ' 

{ay contract for evaluation services became effective approximately the 
same time as the project director and two additional staff were hired. The person 
chosen for/director was one of the'fev; Alasl^an Native^ t9 hold a. doctorate degree 
'and he received faculty rank as well as director position. .Though , discussion on, 
the projjsct h.ad been going on for several years, and the program*- pdlicy council 
had be.en selected and meeting for aTev; months, 'the actual start of the project 
^could be said to have begun at this point, ^ > • /' 

- *■ ' , ' ' 

Logically, I began with the most readily available data, by reviewiiYg the 
proposal on which the grant was-bosed. The final proposal was written in very 
general^ terms leaving the direction of project objectives and activities fo.tTTe 
council. In attempting to flush out this information 1 soon discovered that over ai 
peViod of time^here were ^actually six proposals. First for my own background 
ahd later to^be used for the evaluation .report, I described the proposals in chart 
form as to tHeir main components (origin, needs, objectives, methods, adviaory ^ 
proce^,4mpMcations) and analyzed each propo^sal fur the changes it presented, 
as compared tZ the previous one. This arlalysis saw ah increasing V/idening of 
'the scope and level of intended educational activities plus increasing Native 
invblvQment'ip decision^making . . * * ''^ • • . 



I noted the final proposal wa3 actually two proposals^; onte from the « 
University and one from* the^statewide Native organization. The fact being that 
"as:genBrcU as the statements <^;ere, they were never merged into. one ,^led me'to , ^ 
suspect that -the 'two sponsoring agencies may h-ave had differing goals or goal 
empha'Sis in niiiid. ^ • . 

' /. . ' ' ^. ^ . • ' ' , 

The -only Qther Available data at the tim'p I began were brief minutes of the 
foii.r, previous monthly policy council meetings. Given: the vagueness of the 
proposal; the responsibility left ta,the council to set directions; the^iinique aspect 
of Uni\^ers-ity Native cooperation^on this council; suspected differences of 
agendas between these g^ups; and the limited nature of theerecoi-ded minutes; 
I made an immediate^decision that my first major expenditure of evaluaticn effort 
should bemused to attend and observe council meetings,. at least'nan.tiLthe project 
. got off 'the ground . - 

^ After attending several- co^taciLmeetings it became obvious to me that"the 
project vvas^m^^ing^very §lowly. Alsp, from informal conversations, I felt there^. 
wa3 general dissatisfaction about the^pfogram^s* progress among councii and staff. 
iHawever, such feelings were riot; being'openly discussed at the council meetings. 
Therefore, my second.decision.^/as^o informally interview *each member of the • 
council and otaff , as to what they thought were the project goals and as to.their . 
opinions as to the progress to date.. On an individubl basis, I found', among other 
things, that councirmdmbers freely criticized the staff foJr not* taking mote^ 
leadership' while the^staff criticized the counciLfo;* not setting moire consistent 
Sind clear, directions . , ' * • - - - ■ ' 

At this point, five months after being hjr^d, I submitted my first process 
evaluation report. It consisted of: . (1) the ev^JuStlon overvieAA/' submitted prior 
to contractir\g;''{2) .the analysis of proposals, (3) an analysis of the council 
•minutes as to frequency of type of business conducted,. and a topical and^chrSno- 
logical ordering of motions, (4) unidentified ititeryiew comments, grouped by 
subject (goals, the council, the^university , the staff , ^ctivities^ administrative 
arranqement^^ and evaluation) , (5) the evaluator's. summation pf each section, 
and observations concerning the council, staff, and the* evaluation. An 
introductici/i requested the ?ouncil consider this data as information for clecision 
making, nbted that it was presented in a^way which allowed the council to draw , 
its own dondlusions, anc? added thatjhe evaluator attempted to keep her opinions 
Confined to the last chapter. -In addition, the report was presented as a "fiVial 
draft" and the council was strongly urged to comment individually or collectively 
as a final chapter for. the report. * 

I had expected to lead the council into the rather bulky evaluation report 
.wit^i an oral presentation using flip charts at jhe tune council meeting. ^With 
advice from & management co.nsultant, I, had' planned to present, in outline form; 
a ratio^nale for each sectiort, telling what I had done and why, ralher than 
' presenting any.datti, ^nd doing this before handing over the report. However, ' 



June is the beginning of fishing season and it v/as the only council meetirig q&fclled . 
* that did not fecei^^e th§ necessary quorum to constitute an official meeting. With 
the July meeting cancelled becau'se of v'acations,. J decided to mail the report with 
a cover letter mtrddueing it, rather than wait two mOjjths tp present*it at the next 
meeting. - * ' • . - 

Up to this i^oint I felt.rather^satisfie^ with my initial efforts at process 
evaluation. However, the'bubble soon burst, as the report did not precipitate 
discussion. among counpil menibers af the next meeting,^ and I only recelved^tv/o ^ 
rather superficial written comments, .even after a follow-^up letter. No 'change in 
djieration and iittle-progress was evidence me for several months thereafter. 
.Paring a second round'of individual interviews, more structuied than the first, 
I found most persons Tiad read some of the report, and some riad'even considered 
it interesting and usefu.Lin terms of confirming some cf the^lr own thoughts^ though 
^there was little agreement about which sections' were or were not .considered useful. 
I nowieel the report was not as^usefjal as I had at first hoped' it would be, -nor as * 
much t)f a failure as I at first thought it had bpen . ^ * 

, ^he council finally did take corrective actioi^nd rea$signed the director 

^ in December. •The latest rounds of intbrViews leadsme to<believe that the most 

me^pingful use made of xiie flrst process evaluation w.as as an orientation docament 
\for the new director. • ^ . , 

Most recently, the project has been actively involved in promoting a new 
rural delivery system vvdthin the University. A comprehensive plan .was first 
proposed by the Native brganizations, then^ sponsored by the protect and is now 
^n acc^jpted part of the University.' s program budget currefitjly under negotiation 
with the legislature . ^ ^ ^ * . • 

. (Xhis description has been heceasarily brief, leaving out many details and 
related events. Hopefully, it will provide adequate* background for the points 
which follow.) . v. . * 

Some Project-Specific Observations 

' *^With hindsight, there are som^e characteristics^ of this particular projecc 
which have ^^mplications /or a process evaluator, and I feel need to be noted before 
""making more general comments ^. 'The project is not federally fu^nded, was purposely 
^vague in its proposal, and ^oes not have to meet any specific evaluation require- 
ments V It is 'unique therd* are no programs to compare it to, historically or 
contemporaneously. The program began at the highest organizational levels 
possible and was thereforeiinstantly visable, Given*the political agendas of those * 
involved and its vifeability, the project needs to succeed, or cannot easily admit 
to failure. Hoy/ever, -it is hard to say, what successful might n^an, as differing' 
.goals would suggestMiffering criteria. As a matter of fact, the\e^^ ambiguity of 



goals allowed the project lo come into being. 5omfe might say Its very existence 
in. terms of the prqcess of bringing together two divergent 6oSimumties is in 
itself succes:5. * > . 

Within this politic^ally sensitive and programaticaily vague, setting, I was 

hired. as an evaluator by the progracn policy council rather than project staff. 

One problem with this position vvas that I hrave only suggestive influence rather 

than- direct control over data collection for staff planned activiQes, which turned^ 

out tOjbe^o influence at all . ,AlsO) I Vv^as at that time a nev; person to the state 

and a Junior faculty member / On the council, v7hoae process I am to' evaluate,. « . 

sit^ fny direct supervisor, the' executive vice-president of ^r.y university a'n(!itvvo 

of the most powQnful Native leaders in the state.* Because of this last fact, I am 

not sUre whether the lack of reaction to'oiy first Evaluation report was: (1) a 

questidri of time and timing; (2) siriiply a ladk of influence or (?) the overriding 

,impox:tance of politics to data-b^3ed decision considerations. 

• • • f , 

The 'Second evaluation report is yet to be written. From these early experi- 
ences, I can predict it yv'ill be shorter 'in length and more judgmental in character 
thSfi the f;rst. - , • * - ^ ^ \ ^ 

Some^General Comments on Process Evaluation 

^ ' ^ ^. : ' • . ' 

I would litce to use-my experience to generate some more, general thoughts • 
on process evaluation. These comments are in no* particular order of importance 
• or anything 'else. . ; ' "\ . ^ ' * 

. (1) The- position the process evalu^^tor holds^n relation to ^he^ project staff' 
t / and to those setting policy i^ all importanL The level at which this 

person is tb'operate, including amount of control over recoird keeping 
and inflijencG^^ipon reqviired reporting, i^ofte^ only hazily defined, 
in both process evaluation models and in actual practice. There is also 
judgment tfi be made as to whether this function should be an .internal 
• or-external one, or both. • 

(2) A process ^valuator is probably most necessary wh^n a project is truly 
l^reaklng new .ground and most unsure of its direction. This same 
lack-of definitive, program plan of cour-^e also presents the evaluator 

^ • with* a most difficult situatioh in which to work. 

J[3) Process evaluation methodologies give little guidance afe to which 
. results to report arid how tcfgo about doing it. The question of with 
whom to sharQ what information and at what poirjt in time is seldom 
raised but.is ?ilm'ost always a perplexing problem for a process 
evaluator. ' 

(4) The situational uniqueness 6f.v;hat is describe(j[ in process evaluations 
> usually voicls any contribution toward progidm replication. How.ever, 



process evaluation studies should be Uf^efuhfor theit contribution to 
expanding methodology. For example, having sHown, a' student a 
description of how transactional evaluation was used succ'essfullynn 
one crisis* situation, she was persuaded to try it on her own, whereas 
just reading*' about the methbdology might not have been as convincing . 

(5) More needs to be written orl the use of proces^ evaluation in conflict 
situations — ^whether it is possible and, if so, how it can be done ^ 

without jeopardizing the integrity of the-e^/aluator; and perhaps mOrQ 
importantly, his or her ability to continue in an assigned role. - 

(6) Educational program pr'ocess evaluaiio^i is usually viewed aq a fidld or 
activity in isolation. It would seem more* useful to attempt to relate it ^ 
to other fields, suQh as management consulting, -with which process 
evaluation shares some com'mbn interests and methods of operation, 'but 
which dre longer established *as a legitimate practice.. The parallels . 
between management co^nsulting and process evaluation ssem espeoially 
evident when a ^frogram falters at the implementation stage. of develop- 

• , ment. t - . • ^ , • » h _ 

(7) Proi^ess •evaluation normally assumes the existence of, a program plan. » 
Evaluators should ^onsider Whether thjis plan'fs to viewed as d guide, 
or contract and, .if to ?ome extent -a plan is, considered flexible, how the 
pr-^ess evaluator relates to project staff' on'this rr\atten. ^ ^ 

-In my opinion, evaluation requirement? of .categorical federal programs 
in terms of pre-specified obje.ctive^ have led a constructed view of 
program plans and defensive ev^Uati^ns. Such evaluations tend to be 
of liftlc use for impro^ving- programs and run great risks of rai§sing 
unintended or unanticipated effects. ^ ' « 

i * 

(8) * What are the ethics^ or advisability of accepting a contract to do a * 

•process evaluation *with'lf*ie details unspecified? Js tfhis a desirable>or 
undesirable situatfon for an evaluator? If one accepts such a contract, 
hovv does one iden^fy a plan and ta whom, if anyo^ie, n"eed it be 
> communicated?/ ' * . . » * • 

(9) Finally, are there some^^rogram^ that should not be evaluated, 
especially during their:, conduct as in a process evaluation? Evaluation, 

J' particularly that of a formative, nature, attributes a rationality to the 
f decision-making process which may not be realistic in all cases. Proj-ects 
that are highly political in nature mig-^jt work more effectively wittiout 
program evaluation as we now know it; or,j^it is more to. the point to 
suggest we need to rethink and expand our concept and jr*ethodology of 
process ev/aluation so^as to make Kmore usable for a greater variety 
of educational programs . ^ ^ V ' 
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